Field of the Invention
The present invention relates to an image processing apparatus that performs image restoration processing, a method of controlling the same, and a storage medium, and more particularly to a method of correcting a degraded image using image restoration processing.
Description of the Related Art
In general, when obtaining an image by photographing an object using an image pickup apparatus, such as a digital camera, the image is significantly degraded e.g. due to aberration of an image pickup optical system (i.e. the image is blurred). A blur component of an image is generally caused by spherical aberration, coma aberration, field curvature, astigmatic aberration, or the like, of the image pickup optical system. In an aberration-free state without any influence of diffraction, a light flux from one point of the object converges to one point again on an image pickup surface of an image pickup device. On the other hand, if any of the above-mentioned aberrations exists, light, which should converge to one point again on the image pickup surface, diverges to generate a blur component on an image thus formed.
The blur component generated on the image is optically defined by a point spread function (PSF). Although an image which is out of focus is also blurred, here, a blur of an image caused by aberration of the image pickup optical system even if the image is in focus is referred to as the “blur”.
As for color bleeding on a color image, color bleeding caused by axial chromatic aberration of the image pickup optical system, spherical aberration of color, and comatic aberration of color can be referred to as different manners of blurring dependent on wavelengths of light. Further, as for color shift in a horizontal direction of an image as well, color shift caused by lateral chromatic aberration of the image pickup optical system can be referred to as positional shift or phase shift caused by different image pickup magnifications dependent on wavelengths of light.
An optical transfer function (OTF) obtained by Fourier transform of the above-mentioned PSF is a frequency component of aberration, and is represented by a complex number. An absolute value of the optical transfer function (OTF) (hereafter, the “optical transfer function” is simply referred to as the “OTF” as deemed appropriate), i.e. an amplitude component is referred to as the modulation transfer function (MTF), and a phase component is referred to as the phase transfer function (PTF).
These MTF and PTF are frequency characteristics of the amplitude component and the phase component of degradation of an image caused by aberration, respectively. The phase component is expressed as a phase angle by the following equation (1). Note that Re(OTF) and Im(OTF) express the real part and the imaginary part of the OTF, respectively:PTF=tan−1{Im(OTF)/Re(OTF)}  (1)
The OTF in the image pickup optical system degrades the amplitude component and the phase component of an image, and hence in the degraded image, points of the object are asymmetrically blurred e.g. in a case where the degradation is caused by comatic aberration. Further, in a case where the degradation is caused by lateral chromatic aberration, the image formation position is shifted due to differences in image formation magnification between optical wavelengths, and when the light is received as the RGB color components according to spectral characteristics of light reflected from the object, this causes different image magnifications between the color components.
This causes shifts in image formation position not only between the red, green and blue (RGB) components, but also between the wavelengths in each color component. That is, the image is diverged by the phase shift. To be exact, the lateral chromatic aberration does not generate simple parallel color shift. However, description below will be given assuming that the color shift has the same meaning as the lateral chromatic aberration, unless otherwise specified.
As a method of correcting degradation in amplitude (MTF) and degradation in phase (PTF), for example, a method of correcting degradation using the OTF of the image pickup optical system is known. This method is referred to as image restoration or image recovery. In the following description, processing for correcting degradation of an image using the OTF of the image pickup optical system is referred to as image restoration processing or simply restoration processing.
Now, the outline of image restoration processing will be described. Let it be assumed that a degraded image is represented by g(x, y), the original image is represented by f(x, y), and the PSF obtained by performing inverse Fourier transform on the OTF is represented by h(x, y). In this case, the following equation (2) holds. Note that * represents convolution, and (x, y) represents coordinates on the image.g(x,y)=h(x,y)*f(x,y)  (2)
When the equation (2) is converted to a frequency-based form by Fourier transform, this gives a form of the product, on a frequency-by-frequency basis, as represented by the following equation (3). Note that H represents a result of Fourier transform of the PSF, i.e. the OTF, and G and F represent results of Fourier transform of the degraded image g and the original image f, respectively. Values of (u, v) represent coordinates of a point on a two-dimensional frequency surface, i.e. a frequency.G(u,v)=H(u,v)·F(u,v)  (3)
To obtain the original image from the degraded image obtained through photographing, it is only required to divide both sides of the equation (3) by H, as represented by the following equation (4):G(u,v)/H(u,v)=F(u,v)  (4)
By returning F(u, v) in the equation (4) by inverse Fourier transform to a real surface, it is possible to obtain the original image f(x, y) as a restored image.
Here, assuming that a result of inverse Fourier transform of 1/H in the equation (4) is represented by R, by performing convolution processing on the image on the real surface, as represented by the following equation (5), it is possible to similarly obtain the original image.g(x,y)*R(x,y)=f(x,y)  (5)
R(x, y) in the equation (5) is referred to as an image restoration filter. The actual image has a noise component, and hence if the image restoration filter generated by the reciprocal of the OTF is used as mentioned above, the noise component is amplified together with the degraded image, and as a result, it is impossible to obtain a good image.
To prevent the noise component from being amplified, for example, there has been proposed a method of suppressing a restoration rate of high-frequency components of an image according to an intensity ratio between the image and noise, as in the Wiener filter. Further, as a method of correcting degradation of an image, caused by a color bleeding component, there has been proposed a method of correcting the color bleeding component by correcting the above-mentioned blur component such that the amount of blur is uniform for each of color components of the image.
Incidentally, the OTF changes according to the photographing state, such as a state of a zoom position, and a state of an aperture diameter of a diaphragm. Therefore, the image restoration filter used in image restoration processing is also required to be changed according to the photographing state. For example, in an endoscope for observing an inside of a living human body, there has been proposed a method of eliminating a blur of an image in a range outside an in-focus range of an image pickup section, using the PSF according to a fluorescent wavelength (see Japanese Patent Laid-Open Publication No. H10-165365). In this method, since the fluorescence is weak, an objective optical system having a small F-number is required. However, if the objective optical system having a small F-number is used, a depth of focus becomes shallow, and hence an in-focus image is obtained by performing image restoration processing for a range in which the object is out of focus.
As described above, image restoration processing is performed on an image obtained through photographing to thereby correct the above-mentioned various types of aberration, whereby it is possible to improve image quality. However, in performing photographing, the photographing state and the state of the image restoration filter do not always optimally match. For example, when photographing a three-dimensional object, such a problem occurs.
In the image pickup apparatus, photographing is performed by focusing on one surface of an object space using auto focus or manual focus. In doing this, in a case where the object is three-dimensional, the object distance is different depending on the angle of view. An in-focus object is relatively sharply photographed, but an out-of-focus object is photographed with an amount of blur dependent on the distance. In a case where information on the object distance is acquired only as to an in-focus point, an image restoration filter optimum for each angle of view in this object distance is selected or generated for use.
On an image after image restoration processing, the image restoration filter is optimum for an in-focus object, and hence it is possible to obtain desired sharpness. On the other hand, the image restoration filter is not optimum for an out-of-focus object, and hence although some effect of restoration is obtained, the image is still blurred.
On the other hand, it is conventionally known that a degree of blur dependent on the object distance produces excellent effects in expressing three-dimensionality of an object or expressing an object being watched in isolation from its background. For example, there is a method of expression in which by using a telephoto lens with a shallow depth of field, an image is expressed such that a main object is in focus and the background is intentionally blurred. In this case, also on the image after image restoration processing, it is desirable that the in-focus object is made sharper, and the out-of-focus object remains still blurred, and blurring expression is performed by using the above-mentioned image restoration method.
However, if the out-of-focus object is subjected to image restoration processing using an image restoration filter which is not optimum for the distance of the out-of-focus object, coloring sometimes occurs on the image. Note that the term “coloring” refers to a defect that a color which is not included in the object is found in the image after image restoration processing because a relationship of blurring between the respective color components on edge portions of the out-of-focus object is different before and after execution of image restoration processing.
Further, such coloring sometimes occurs not only in photographing of a three-dimensional object. More specifically, coloring occurs irrespective of whether or not the object is in focus, if a state of aberration in the actual photographing state and a state of aberration targeted by the image restoration filter are different e.g. due to manufacturing variation of the image pickup optical system or variation of spectrum of a light source in photographing.
As a method of suppressing the coloring described above, there has been proposed, for example, a method of correcting the color of an image after image restoration processing based on color information on the image before being subjected to image restoration processing. In this method, a change in color, caused by image restoration processing, is determined for each pixel of the image to thereby suppress coloring caused by image restoration processing.
For example, there has been proposed a method of correcting a signal value such that an amount of color difference is reduced, in a case where the color difference in an image after being subjected to image restoration processing becomes larger than before being subjected to image restoration processing (see e.g. Japanese Patent Laid-Open Publication No. 2010-86138).
As described above, by performing image restoration processing on an image obtained through photographing to reduce coloring which occurs e.g. on an image of an out-of-focus object, and correcting various types of aberration, it is possible to improve image quality.
However, as described hereinafter, when coloring suppression processing is performed according to a color difference before and after being subjected to restoration processing, color tone of the object in the image is sometimes changed or made inaccurate. Further, color tone is changed also depending on characteristics of the image pickup optical system and the sensitivity (ISO sensitivity) of the image pickup device.
However, the method described in Japanese Patent Laid-Open Publication No. 2010-86138 does not address these changes, and hence it is difficult to properly suppress coloring caused by image restoration processing.